A Field Critique of the 3-Year Pilot Test for the CUSTOMER Recreation Visitor Survey1

From 1990 to 1992, the USDA Forest Service implemen­ ted a 3-year pilot test of CUSTOMER, a standardized nationwide recreation visitor survey. Intended as a partnership between the agency’s Research and National Forest System branches, CUSTOMER has been a limited success to date. By the end of 1993, nearly 20,000 recreation visitors had been interviewed in more than 35 different sites, including 15,800 National Forest visitors from Alaska to Puerto Rico. Resulting data have not been fully analyzed to date nor been made available for recreation researchers, managers, or the general public. The results of a telephone interview of past CUSTOMER users is presented regarding their evaluation of the recreation visitor survey. Inherent conflicts within the multiple goals of CUSTOMER are examined, as well as insufficient funding and sponsorship, methodological compromises, and availability of alternative means for data collection that may be curbing a wide-spread demand for the survey in its present form. Six general recommendations are offered to help improve identified problems in CUSTOMER. The need to better understand the recreation use and users of the National Forest System has been evident since the passage of the Forest and Rangeland Renewable Resources Planning Act (RPA) in 1974 and the National Forest Management Act in 1976. Since the passage of these acts, the practical and theoretical goals of studying recreation have evolved from the collection of relatively simple quantitative measures to more complex qualitative measures of user behavior (Manning 1986). As suggested in the 1989 USDA Forest Service RPA Assessment, and by others including the Office of Technology Assessment (1992), further effort is needed to collect comprehensive recreation visitor information that will enable recreation providers to improve the quality of recreation planning and management nationwide (Cordell and others 1990). The importance of studying Federal “customers” was most recently reinforced in an Executive Order signed by President Clinton on September 11, 1993, which charged all executive departments and agencies (including the Forest Service) with “ensuring that the Federal Government provided the highest quality service possible to the American people” (Clinton 1993). Specifically, the Order called for agencies to “survey customers to determine the kind and quality of services they want and their level of satisfaction with existing services.” An abbreviated version of this paper was presented at the Second Symposium on Social Aspects and Recreation Research, February 23-25, 1994, San Diego, CA. Social Scientists, Chugach National Forest, 3301 C. Street, Suite 300, Anchorage, AK 99501 One of the more ambitious efforts in recent years to improve understanding of recreation visitors nationwide, the Public Area Recreation Visitor Study (PARVS), was begun in 1985 by the USDA Forest Service’s Southeastern Forest Experimental Station (SEFES). That Station has proposed to supersede PARVS by the more comprehensive Customer Use Survey Techniques for Operation, Management, Evaluation, and Research (CUSTOMER) recreation survey. In response, the Office of Management and Budget (OMB) issued approval in 1990 for a 3-year pilot test (1990-92) of CUSTOMER to evaluate the methodology and potential effectiveness of CUSTOMER. During the past 3 years, CUSTOMER has been one of the few large scale studies of the Federal land recreation visitors and the only one to be conducted by a Federal agency. This paper offers a field-level review and discussion of the 3-year pilot test of the CUSTOMER recreation visitor survey. After the achievements of CUSTOMER during the 3-year pilot test are summarized, the results of a telephone interview of personnel from sites that have utilized CUSTOMER are presented. Finally, three general issues associated with the CUSTOMER survey are discussed, as well as six recommendations for improving CUSTOMER (or other subsequent recreation survey projects). CUSTOMER Achievements and Current Status During the 3-year pilot test (and 1993) on selected Federal lands nationwide, CUSTOMER was successful in generating a wide range of recreation visitor data using a combination of on-site interviews and mailback question-naires. As of December 31, 1993, CUSTOMER had been implemented at more than 35 sites (defined as a unique combination of a specific administrative unit and recreation season) in 17 States, including 28 sites administered by the Forest Service, 7 by the USDI Bureau of Land Management, and one each by the USDI National Park Service and Tennessee Valley Authority. (CUSTOMER has also been used as a component of several other economic value studies around the Nation.) More than 19,850 recreation visitors were interviewed on site, with 43 percent also completing and returning one or more additional mailback surveys. Some 80 percent of the total interviews (15,800) occurred within National Forests in all major geographic regions although the regions were sampled disproportionate to the amount of recreational use they received (USDA Forest Service 1993). Within the National Forests sampled, visitors engaged in all major categories of recreation activities (as defined by the Forest Service’s RIM [Recreation USDA Forest Service Gen. Tech. Rep. PSW-156. 1995. 121 Information Management] and RRIS classifications) were interviewed, although again, disproportionate to use. CUSTOMER on-site interviews yielded information on selected demographic characteristics of visitors, including respondent sex, age, race/ethnicity, education level, employment, physical and learning impairment, household type, household income, group type, and group size. Selected trip characteristics of respondents included residence or market area; trip purpose; information sources for trip planning; primary destination; repeat versus first-time visitation; distance and hours travelled; length of stay; nights on site; visits during last 12 months; year of first visit; reasons for choosing a site; and identification of substitute sites. Finally, the on-site interview collected visitor attitudes about selected site-specific recreation planning and management issues. Following the on-site interview, visitors were given mailback questionnaires asking them to rate the importance of generalized setting attributes for both ideal settings and those at the site. Visitors also rated their satisfaction with those same attributes. In accordance with the work of Martilla and Hames (1977), importance and satisfaction ratings were subsequently combined to suggest appropriate general management responses. An optional second mailback questionnaire captured visitor trip and equipment expenses that could be utilized in the IMPLAN economic input-output model to indicate the economic impact of recreation use on surrounding counties. A Field Evaluation of CUSTOMER Despite the cost and amount of effort involved in the development and application of CUSTOMER during the 3-year pilot test, to date the survey’s implementation, effectiveness, or use by the sites that have implemented CUSTOMER have not been systematically evaluated. Accordingly, for this paper one or more representatives from each site that had contracted for CUSTOMER during its 3-year pilot test were interviewed over the telephone by the authors between December 1993, and January 1994. The purpose of the interview was to evaluate CUSTOMER based upon the experiences of the sites. Two types of site representatives were defined and contacted: (1) “planners,” or those involved in initiating and structuring the customer survey for the site, and (2) “users,” or those who now apply the results of CUSTOMER data in the ongoing planning and management of the site. A total of 47 site representatives were identified and interviewed. (Four representatives had retired subsequent to their involvement with CUSTOMER and were not contacted.) The responses of each site representative were weighted equally in the analysis of the interview results although in several cases an individual was involved as both a planner and a user at the site, or was involved in multiple implementations of CUSTOMER. The responses of planners and users were compared in order to detect differences in their expectations for CUSTOMER and its subsequent performance. And responses for the years 1990, 1991, and 1992 seasons were compared in order to recognize improvement or maturation of the CUSTOMER project. And the Forest Service was compared with other agencies in order to suggest whether CUSTOMER was found more or less suitable by Forest Service site representatives as a group. Expected Use of CUSTOMER Data Site representatives were asked to indicate whether they had expected to use CUSTOMER data for several advertised uses of CUSTOMER. The most expected and most frequently used application of CUSTOMER data reported was for uses associated with new investments and capital improvement projects (89 percent), followed by site planning (83 percent), and forest planning and plan revision (77 percent). The application with the least expected and actual use was training (17 percent). Other possible expected uses included marketing and publicity, environmental impact statements and assessments, congressional data requests, budgeting and staffing, and conflict resolution. Up to 35 percent of the site representatives reported that they had not yet had an opportunity or need to use the data for one or more of the expected purposes. Many expected that they would eventually use the data. Usefulness of CUSTOMER Data Site representatives were asked to rate 10 specific areas of CUSTOMER data in terms of usefulness for meeting their planning and management needs. Using a 5-point scale ranging from “very good” (5) to “very poor” (1), the highest ratings were given to visitor demographic profiles (4.2), trip profiles (4.1), and visitor setting preferences (4.1). The least useful data were associated with spending and economic impact (2.9). Other possible areas of data included activities engaged in, visitor setting satisfactions, values and markets, special issues, and willingness-to-pay. The mean ratings for several areas of information differed significantly (p <= .05) by agency and by year. CUSTOMER Implementation Using the same scale, site representatives were asked to rate how well nine different stages of CUSTOMER were implemented at their sites. Volunteer training and interviewing (4.2), pre-survey consultation (3.9), and on-site set-up and sampling (4.0) were most highly rated. The lowest rated implementation stage was post-survey consultation (2.4). Other implementation stages included final data report, presentation of results, mid-survey consultation, and contract administration. The mean ratings for several implementation stages were significantly different (p <= .05) by year, the highest ratings given to CUSTOMER implementation in 1991. 122 USDA Forest Service Gen. Tech. Rep. PSW-156. 1995. Future Use and Recommendation Site representatives were asked if they would consider conducting another CUSTOMER survey at their site within the next 3 to 5 years. The most common responses were “No—no need to update data” (20 percent), “Maybe—survey funding is uncertain” (17 percent), and “No—would seek other sources for data” (15 percent). Thirteen percent indicated “No—not satisfied with CUSTOMER” and only 6 percent indicated “yes—satisfied with CUSTOMER.” The frequency for the “No—not satisfied with CUSTOMER” response differed significantly (p ≤ .05) by agency, with Forest Service site representatives less likely to be unsatisfied. Most site representatives responded “Yes—without conditions or reservations” (47 percent) when asked if they would recommend CUSTOMER to other National Forests, parks, wildlife refuges, or Bureau of Land Management districts. Forty percent indicated “Yes—with conditions or reservations,” but 13 percent responded “No—would not recommend CUSTOMER.”